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APOLLO EXPERIENCE REPORT 
FLIGHT ANOMALY RESOLUTION 
By John D. Lobb 

Lyndon B. Johnson Space Center 
SUMMARY 


The frequency of the Apollo missions demanded that flight anomalies be quickly 
identified and resolved so that prompt corrective action could be taken. Analyses of the 
data for problems and anomalies had to be compressed into a relatively short time 
frame. 

Timely resolution of flight anomalies was achieved during the Apollo Program 
so that corrective actions necessary for crew safety, mission success, and program 
success were completed during each mission and between missions. This success was 
largely due to a team of engineers who understood the details of hardware design, test 
history, and operational characteristics of their respective systems. A working rela- 
tionship that ensured quick responses and concurrence of this team with program 
management and flight operations personnel contributed equally to program success. 

The analysis team had access to the support of any other specialists or facilities needed 
to the extent justified by the nature of the particular problem. Team understanding of 
the mechanism associated with the cause of the problem was required for resolving 
each anomaly. Complete understanding of an anomaly often required postflight data 
analysis, crew debriefings, and testing of spacecraft hardware. 

The efficient documentation system established a definite point on which to focus 
direction and continuity of effort for more effective anomaly resolution. The flexibility 
of some of the documentation provided continuous updating by the analysts and ensured 
a common understanding of the current status of each system problem by the analysis 
team and by program management and flight control personnel. 

Of course, the complexity of the spacecraft configuration and of the complement 
of experiments taken on the lunar -landing missions increased the probability of a wide 
variety of anomalies. The same general procedure was used to properly identify and 
understand each problem, analyze the cause, and determine the proper course of action. 
Examples of actual flight anomalies illustrate the general types of causes identified, 
the levels of analysis required, and differences in the nature and extent of the correc- 
tive actions justified. 



INTRODUCTION 


Flight anomaly resolution was the mission evaluation task that covered space- 
craft and experiment problems and deviations from expected or predicted performance. 
This task included not only the resolution of true failures and malfunctions, but also the 
resolution of items that appeared to be problems or discrepancies at first but that, on 
further investigation, proved to be incorrectly identified as anomalies. Thus, the term 
’’resolution, " as used in this report, includes the proper identification of the anomaly 
as well as the determination of the cause and the proper course of action to be taken 
both during the mission in progress at the time and before subsequent missions. 

The flight anomaly resolution system used during the Apollo Program was an out- 
growth and further development of the mission evaluation procedures used on the pre- 
vious manned space flights of Project Mercury and the Gemini Program. Mission 
evaluation ensured that maximum information was obtained from each Apollo flight for 
use in managing and planning later flights and future programs. The information was 
also provided to the scientific community and to the public so that they could be well 
informed on the activities of U. S. space exploration. 

It was necessary that anomaly resolution be completed in a timely manner for 
the interim in-flight course of action. Just as important was the need for a time frame 
that would enable completing the proper corrective action for subsequent missions with- 
out requiring changes in the scheduled launch dates for lunar -landing missions, which 
were limited to specific lift-off times and calendar dates for each lunar exploration site. 

This report contains a discussion of the primary characteristics of the success- 
ful resolution of Apollo flight anomalies. Typical experiences have been used to 
illustrate how the system was actually implemented. In addition to the examples of 
flight anomalies discussed in the body of the report, extracts from actual anomaly 
resolution reports for four problems are presented in the appendix. These extracts 
have been included to illustrate that changes from issue to issue of the reporting system 
reflect the progress made in resolving anomalies. The four anomalies selected reflect 
different combinations of causes, sources of data for analysis, corrective actions, and 
reporting histories. 


PERSONNEL 


Essential to the successful performance of anomaly resolution was the makeup 
and interaction of the mission evaluation team. Each of the many disciplines involved 
was represented on the team by a group of specialists managed by a NASA team leader 
who was also a specialist in the same discipline. Each specialist had a thorough under- 
standing of system characteristics, operations, and limitations gained from experience 
with his particular systems from initial design through development and testing of the 
hardware. Team members were selected from NASA and contractor personnel. 

Each team leader was the analysis manager for anomaly resolution within his 
particular range of responsibility. For example, the Apollo 14 team included an 
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analysis manager for each of the following: telecommunications, crew systems, elec- 
tronic systems, propulsion and power, guidance and control, structures and mechanics, 
thermal control, flight crew support, trajectory, and lunar surface experiments. 

The responsibilities of each team leader included coordination with the other team 
leaders to ensure that any solution or recommended course of action would not jeopard- 
ize other systems. The team leaders reported to an evaluation team manager who 
was responsible for the overall operation of the team, reviewed and integrated the inputs 
from the team leaders, and communicated with the Mission Control Center as the single 
point of contact. In addition, an anomaly engineer was responsible to the team manager 
for following through on each anomaly to completion of the resolution. There was an 
anomaly engineer for command and service module problems, one for lunar module 
problems, and one for experiments and Government -furnished-equipment problems. 
Continuity to completion of the activity was enhanced because some team members in 
each discipline were also responsible for the detailed implementation of hardware 
changes required for subsequent missions. The mission evaluation team provided 
direct support to the flight control organization in the Mission Control Center for in- 
flight resolutions and to program management for resolutions requiring action on the 
ground before subsequent missions. 

The premission planning and training of flight operations personnel with the crew- 
men covered many possible failures and malfunctions of systems and components. 

These failure modes and corrective contingency procedures were thoroughly exercised 
during simulations. Nevertheless, it was not surprising that unpredicted problems and 
discrepancies were encountered with a complex configuration such as the spacecraft 
and the complement of experiments taken to the Moon. The prime tasks for the mission 
evaluation team were to properly identify and understand these problems, to perform 
the investigations and analyses required, to determine the causes, and to recommend 
the proper courses of action. 

In support of the team effort at the NASA Lyndon B. Johnson Space Center (JSC) 
(formerly the Manned Spacecraft Center (MSC)), the prime contractors maintained 
similar teams of specialists at their facilities during the mission for analyses, tests, 
and related activities as they were required. These teams also reported to the mission 
evaluation team management at MSC. 

Many times during Apollo flights, the support team personnel at the prime con- 
tractor facility provided test information that was invaluable for understanding a prob- 
lem and for determining the best course of action. For example, during lift-off of the 
Apollo 12 spacecraft, a lightning discharge resulted in tumbling of the inertial platform 
and loss of inertial reference. The support team at the Massachusetts Institute of 
Technology in Cambridge, Massachusetts, performed a test simulating the condition in 
an attempt to understand how the potential discharge had caused the platform to tumble. 
It was important to quickly verify that the platform had not been damaged. A prompt 
response was received by the evaluation team, and the mechanism that had caused the 
conditions in flight was determined. 

The choice of team personnel for Apollo flight anomaly resolution was a major 
contribution to the success of the missions. Many of the analysis managers continued 
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in that capacity for the entire program and, thus, provided continuity to the anomaly 
resolution and corrective action effort as well as exceptionally good intrateam under- 
standing. The team manager, who was also chief of the MSC Apollo Spacecraft Pro- 
gram Office Test Division, led the mission evaluation and anomaly resolution operation 
for the Apollo Program, beginning with the Apollo 6 flight. 


IDENTIFICATION OF ANOMALIES 


The first task of the mission evaluation team was to identify the anomaly. This 
task required the team’s constant awareness of the total system performance as the 
mission progressed. Individuals monitored telemetry data that were received by the 
tracking stations, transmitted to and processed by the Mission Control Center, and 
displayed on closed-circuit television receivers in the Mission Evaluation Room. In 
addition, the air -ground voice communications and television coverage of crew activi- 
ties were closely monitored. Selected telemetry data, which were not on the television 
displays, were provided to team members in the form of computer tabulated printout 
copies several hours after transmission for analysis of system performance. Identifi- 
cation of flight anomalies continued after completion of the mission through crew 
debriefings and additional individual conferences between specialists and the crewmen, 
results of the analyses of in-flight photography, and results of the analyses of reduced 
data from the mission. 

Identification of anomalies for the lunar surface equipment operating at the five 
Apollo science stations will continue for the remaining data -gathering lifetime of each 
installation. When indications of a problem appear in the telemetry data, various 
troubleshooting command sequences are transmitted to the affected station to help 
identify the possible anomaly. Initial indications sometimes are evident in the real- 
time data; at other times, the anomaly may not be evident until the more detailed data 
processed by the principal investigator have been analyzed several months later. 


I n sufficient Ground Test Data 

System performance was usually determined by comparing the flight data with 
performance predictions and with data from ground tests and from other flights. Data 
comparison was also a prime method of detecting anomalies. However, in the most 
difficult identification problems, the available data were not sufficient for complete 
understanding of all normal operating characteristics of the system. 

A typical example of this condition occurred on the Apollo 7 mission. The com- 
mand module entry battery recharging characteristics were below predicted levels 
throughout the flight, and the planned charge levels could not be maintained. Available 
ground test data were insufficient to explain the condition. The condition was duplica- 
ted during postflight tests, and a detailed analysis was made. Results of the analysis 
showed that the charger to battery line impedances, which were not evaluated previously, 
were large enough to cause a significant reduction in the charge voltage applied at the 
battery. Results of the system analysis also showed that the charger output voltage was 
within specifications, but on the low side, and that the characteristics of an experimen- 
tal plate -divider material used in the batteries could result in a condition, at zero g, 
that could limit the recharge capacity of the battery. 
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As a result of the anomaly investigation, chargers with the highest voltage out- 
put were selected for subsequent missions, and integrated systems tests were per- 
formed to establish the overall system characteristics of each installation and, thereby, 
to ensure adequate recharge capability. In addition, for Apollo 11 and subsequent 
missions, the charger was modified for a higher output voltage setting, and conven- 
tional plate -divider material was used in the batteries. Performance on subsequent 
missions was satisfactory. Additional detail on this anomaly is contained in 
reference 1. 


Insufficient Flight Data 

Sometimes, the flight data were insufficient for immediate identification of a 
specific problem. For example, during the Apollo 15 mission, the service propulsion 
system thrust light, located on the entry monitor system panel, was illuminated 
shortly after transposition and docking at a time when no engine -firing command was 
present. Because the problem could not be accurately identified by using available data, 
the crewmen were provided with special troubleshooting procedures to better pinpoint 
the problem. Application of the fault isolation procedures resulted in isolation of the 
malfunction to a short circuit in bank A of the service propulsion system ignition cir- 
cuitry. The results of the next special procedure - to manually initiate ignition for 
the first midcourse correction thrust firing by closing the pilot -valve main A circuit 
breaker - further verified that the short circuit existed on the negative side of the 
service propulsion system pilot-valve solenoids, and also indicated that subsequent 
maneuvers could be controlled by manual control of bank A. Results of postflight 
tests and examination of the returned hardware showed that the intermittent short 
circuit was caused by a freefloating strand of contaminating wire inside the system A 
differential -velocity thrust switch. All switches of this type were subjected to addi- 
tional screening procedures for subsequent missions. 


Obvious Component Failure 

Many anomalies were recognized simply because a specific component was 
obviously not functioning. For example, during the initial deployment of the Apollo 12 
color television camera on the lunar surface, the picture was lost because the camera 
was inadvertently pointed at the Sun. The direct sunlight had caused a light overstress, 
which damaged the image sensor to the extent that a usable picture could no longer be 
attained. For further details, see the appendix and reference 2. 


Crew Observation 

Sometimes, the initial indication of an anomaly resulted from observation of an 
off-nominal condition by a team member or by a crewman, reported either at the time 
the problem occurred or later if the problem was not critical. An example was the 
postflight report by the crewmen concerning the failure of the primary floodlights in 
the command module lower equipment bay during the Apollo 7 mission. Subsequent 
investigation revealed that the lamps had been operated excessively in the dimmed 
mode before flight. For additional information, see the appendix. 
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ANALYSIS 


Two basic techniques were used to determine the cause of a problem and the 
required corrective action. The first technique was the experimental testing of the 
actual or identical hardware under actual or simulated flight conditions or under labo- 
ratory ambient conditions. The use of breadboard systems when required was another 
type of experimental testing available to the mission evaluation team. The second 
technique was analytical, generally in the classical sense. The analytical technique 
included the use of computerized mathematical models, the application of physical laws, 
and the use of established engineering formulas. 

The choice of technique depended on the nature of the problem. The most expe- 
dient approach was sought; however, attention to detail was also of prime importance 
because the analysis had to be sufficiently rigorous to establish the cause and, then, 
to develop and prove the necessary corrective action to prevent future occurrences. 

Of course, corrective action sometimes was unnecessary. 


Extensive Analysis 

The extent of the analysis varied considerably and depended on the significance 
and nature of the problem. Extensive analysis sometimes was required. For example, 
on the Apollo 6 mission, a structural failure occurred in the adapter that housed the 
lunar module during launch and provided the structural connection between the service 
module and the launch vehicle. This adapter had the largest honeycomb structure 
designed and developed for any application to that time. Photographs from mission 
support aircraft and ground-based long -range cameras showed that much of the honey- 
comb sandwich facesheet had separated abruptly from one of the four circumferential 
sections of the adapter during launch. The effects of the separation were indicated on 
many measurements in the command and service module, the lunar module, and the 
launch vehicle. The adapter continued to sustain the required loads, however. 

The seriousness of the problem because of its possible effect on the integrity of 
the space vehicle and impact on the Apollo Program required an extensive analysis. 

The investigation of this anomaly included testing full-scale hardware under dynamic 
and static conditions, performing many experimental tests on smaller test articles, 
and performing extensive structural analyses at various NASA centers and contractor 
facilities. To determine the cause of this anomaly, analyses were made of the struc- 
tural dynamics of the launch vehicle, the dynamic loads of the lunar module, the 
dynamic modes of the adapter itself, the effect of thermal and pressure environments 
during launch, and the assembly techniques and quality control procedures used in the 
manufacture of the adapter structure. 

The investigation was focused first on an understanding of the coupled vibration 
modes and vibration characteristics of the launch vehicle, the spacecraft, and the 
adapter. Results of extensive vibration tests showed that the failures had not been 
caused by vibration. Investigations of the thermal and pressure conditions, which 
included full-scale simulations, revealed that the pressure buildup within the honeycomb 
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sandwich, because of the effect of aerodynamic heating on air and moisture trapped in 
the panel, could have caused the anomaly wherever a weak bond existed in the panel 
structure. 

Analysis of the original ultrasonic inspection scan records of the Apollo 6 adapter 
assembly showed an abnormal condition extending several feet along a panel internal 
structural splice in the center of the region where the adapter failed in flight. Nor- 
mally, whena suspect condition was detected duringthe manufacturing.process, the area 
was X-rayed. If the X-rays confirmed a weak joint or bond, a void, or a misalined 
component, the fault was repaired. Nevertheless, the anomalous indication at this 
location had been overlooked on the ultrasonic inspection scan record during manufac- 
turing. The void and misalined splice indicated by the observed abnormality caused a 
weakness of sufficient size that the facesheet was blown off by internal pressure build- 
up during launch. 

As a result of the anomaly investigation by the mission evaluation team, the 
following was accomplished. 

1. The probable cause was identified. 

2. Similar conditions found on other adapters were repaired. 

3. Vent holes were drilled through the inner facesheet, and cork insulation was 
added on the outer facesheet to reduce the temperature and pressure buildup in the 
honeycomb panels. 

4. Manufacturing assembly techniques and quality control applications were 
modified to avoid repeating the condition on future assemblies. 

5. The fact that the failure was not the result of a basic design deficiency was 
established. 

6. The structural integrity of the adapters for subsequent missions was verified. 


Limited Analysis 


An example of the other extreme in the extent of analysis required was the type 
of anomaly for which no corrective action was taken because of the nature of the 
failure. For example, on the Apollo 11 entry monitor system, an electroluminescent 
segment of the velocity counter would not illuminate. A generic or design problem was 
considered highly unlikely because of the number of satisfactory activations experienced 
previously. A circuit analysis produced a number of failure mechanisms; however, 
there was no failure history in any of the circuits. This anomaly was typical of random 
failures that could occur. The basic design concept of the spacecraft provided for 
alternate procedures or redundant equipment to perform the function; consequently, 
no corrective action was taken for this and similar problems. 
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Returned Flight Hardware 

Sometimes, the analysis and resolution could not be completed until the returned 
hardware had been examined in detail. Examples of dependence on flight hardware for 
anomaly resolution were the failure of the Apollo 12 color television camera on the 
lunar surface and the failure of the primary lower equipment bay floodlights on the 
Apollo 7 mission (ref. 2 and appendix). The floodlights were returned because they 
were part of the command module equipment. However, the television camera was 
returned because of the anomaly resolution requirement. The camera had not origi- 
nally been listed for return stowage. For details of the Apollo postflight testing pro- 
gram, see reference 3. 


Unreturned Flight Hardware 

In general, two types of malfunctioning equipment were not returned. One type 
was hardware that had been deployed and was operating as part of a lunar surface 
science station, such as the Apollo 14 active seismic experiment geophone that was 
producing erratic data (appendix). Testing to resolve the problem was accomplished 
by using breadboard equipment in the laboratory and by transmitting commands to the 
science station on the lunar surface. 

The other type of problem equipment that could not be returned was equipment 
located in the lunar module or in the service module. Oscillation of fuel cell condenser 
exit temperatures during the Apollo 10 and 11 missions was a good example. See ref- 
erence 4 and the appendix. The fuel cell condenser was mounted in the service module, 
which separates from the command module at the time of entry and is not recoverable. 
Extensive review and analysis of the flight data for all previous missions were neces- 
sary because the condition had been observed on several missions. 


TYPICAL CAUSES OF ANOMALIES 


The causes of most anomalies were related to manufacturing quality, hardware 
design, or operational procedures or to a combination of these. 


Manufacturing Quality 

The structural failure of the Apollo 6 adapter was primarily a manufacturing 
quality problem because of the failure to adequately interpret the ultrasonic scan 
record. If the record had been interpreted properly and the fault confirmed by X-ray, 
the correct structural repair would have been implemented. In addition, there had 
been manufacturing process difficulty in properly alining all structural parts when the 
assembly was prepared for bonding in the autoclave. Manufacturing quality anomalies 
also included irregularities such as improperly torqued screws, incorrectly soldered 
joints, and improperly potted components which resulted from deficient manufacturing 
procedures. 
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Hardware Design 

System anomalies caused by hardware design deficiencies sometimes occurred 
because all factors that would influence the performance of the system were not 
included in the design criteria or specifications. Because component and system test- 
ing was based on these criteria or specifications, the deficiency could go unnoticed 
during development and qualification tests and not be detected until actual flight condi- 
tions were encountered. A typical illustration was fogging between the panes of the 
Apollo 7 command module windows caused by deposits from the outgassing products of 
the room -temperature -vulcanizing sealing material around the windows. The design 
of the assembly had not required that the sealing material be cured at the operating 
temperature and pressure. Such a curing process would have reduced the amount of 
outgassing during flight. For further discussion of this problem, see reference 5. 

Other types of design deficiency anomalies were those in which the state of the 
art had advanced or additional knowledge and experience in space had been acquired 
following the design and manufacture of the component or assembly. An example of 
this type was the cracking solder phenomenon that occurred on electronic hardware 
designed and manufactured before 1967. An investigation, performed at that time 
because of problems with spacecraft and launch vehicle electronics, revealed that 
existing design practices for printed circuit board assemblies did not allow for occur- 
rence of thermal expansion of the assemblies during normal operations. Differences 
in the thermal expansion properties of the board materials, coating materials, pot- 
ting materials, and component lead materials caused stress and cracking in the 
soldered connections. The program also resulted in development and publication of 
tested, acceptable design configurations and guidelines for printed circuit board 
assemblies. See references 6 and 7. Many non-mission-critical electronic assemblies 
had been built before that time. Although the assemblies were subjected to additional 
testing and screening, some anomalies did occur as a result of this design weakness. 
For example, during the Apollo 12 mission, a failure occurred in the mission timer 
tuning fork circuit because of a cracked solder joint in the cordwood construction. 
Electrical components such as resistors, capacitors, and diodes had been soldered 
between two circuit boards, and the void between the boards had been filled with potting 
compound. The thermal expansion differential between the potting compound and the 
component leads resulted in solder joint cracks at a board -to -lead connection. Design 
of a new clock, developed for subsequent missions, did not include the cordwood 
construction. 


Operational Procedures 

Procedural problems in operating the various systems and equipment were 
usually easy to correct. An example of a procedural problem occurred during the entry 
and landing sequence on the Apollo 12 mission. A crewman was struck by a 
16-millimeter camera that was dislodged from the mounting bracket near the rendez- 
vous window at splashdown. The camera was to have been stowed before landing because 
design of the mounting bracket did not include retention under landing forces. Designa- 
tion of the camera for stowage in the flight plan and the crew checklist on subsequent 
flights precluded recurrence of this incident. 
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The correction of some procedural problems involved both crew training and 
hardware design changes. This type of correction was required after the loss of lunar 
surface television transmission during the Apollo 12 mission (appendix and ref. 2). 

It was found that high -intensity light inputs were received not only during direct point- 
ing at the Sun, but also during normal pointing as a result of specular reflections from 
objects in the field of view. This problem hastened completion of the development of 
new burn -resistant image sensors. The newly developed tubes were used with success 
on subsequent missions. Operational procedure on later missions was established to 
avoid pointing the television camera at specified light sources. Also, with the addition 
of the lunar roving vehicle having a television camera mount on the Apollo 15 mission, 
remote pointing control from the Mission Control Center freed the flight crew from 
most of this responsibility. 

Another type of procedural problem was illustrated by the short in-flight life of 
the floodlights in the Apollo 7 command module lower equipment bay (appendix). 

Here, the preflight procedures were modified to ensure sufficient floodlight lifetime 
for mission operations. 


DOCUMENTATION 


Documentation provided, in firm and visible form, definitive information for a 
better understanding of each anomaly throughout the resolution process. Clearly 
written explanations, accompanied by illustrations and data when required to clarify 
or back up a point, ensured a common understanding of the status, findings, and final 
resolution of anomalies by management, engineering, and operations personnel. The 
character of the documentation and the scheduling of the documentation also forced 
analysts to organize the work so that the results and conclusions were concurred upon 
by the working groups and reported in an understandable format and in a timely manner. 


During Flight 

During flight, the anomaly resolution activity of the mission evaluation team was 
documented in several different formats, each serving a specific purpose and each 
supporting the other. Communications between the Mission Control Center and the 
Mission Evaluation Room about questions, answers, and specific information concern- 
ing the spacecraft and experiments hardware were documented on a spacecraft- 
analysis/mission-evaluation action request form. This documentation was often the 
written confirmation of concurrent verbal understandings between the two groups, 
sometimes containing additional detail backing up the verbal action. The identification 
of an anomaly was frequently documented for the first time in one of these messages. 

The request forms issued by the mission evaluation team, in the form of ques- 
tions, responses, or information, were signed by the originating specialist, by the 
analysis manager or managers for the system or systems affected, by the contractor 
representative to indicate concurrence from the team at the contractor facility, and by 
the evaluation team manager to indicate that the contents covered all affected systems. 
Copies of these messages were also forwarded by wire to the contractor, and contrac- 
tor inputs to the team at MSC were documented on similar forms. 
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Because the actions covered by these communications were for real-time deci- 
sions, response time was kept to a minimum. Sometimes, the answer was almost 
instantaneous and required close-knit, alert team action. Sometimes, an immediate 
reply required action to achieve a safe standby condition until the results of expedited, 
special ground tests and analysis provided a more permanent, longer range procedure. 
The team handled an average of 325 of these requests for each lunar -landing mission. 

In addition, anomalies were noted briefly in the 2 -hour status reports prepared 
by the analysis managers during the mission. In these reports, the status of each 
spacecraft system and experiment was presented for each 2 -hour period during the 
mission. As a result, a problem would sometimes be described several places in the 
same report because of its effect on more than one system. This report provided addi- 
tional assurance that team members were aware of problems and trends in other sys- 
tems as well as in their own. The daily report was, in essence, a summary of the 
2 -hour reports for the preceding 24 hours. 

As problems were identified, they were documented on a problem -tracking list, 
which included a statement of the problem as it was understood at the time, identifica- 
tion of the responsible analysis manager or managers, and a brief discussion of the 
problem, including the action in progress and the schedule status of the resolution. 

This list, distributed to team members and posted in the Mission Evaluation Room, 
established a definite point on which to focus direction and continuity of effort so that 
anomaly resolution would be more efficient and effective. Items that had been incor- 
rectly identified as problems because of insufficient or misinterpreted information were 
quickly recognized as normal operations, and efforts were directed toward resolving 
the authentic anomalies. 


After Flight 

After flight, the documentation was changed to suit the postmission testing and 
analysis required for anomaly resolution. The problem -tracking list was expanded into 
the Problem and Discrepancy List reports, which included separate documentation for 
each anomaly. Examples of the Problem and Discrepancy List format for individual 
problems are included in the appendix. The extract from the Apollo 14 mission is an 
example of the final format for the program. 

The Problem and Discrepancy List was updated frequently as the fast -moving 
resolution process progressed. Statements in each issue of the report were subject to 
change as the analysis continued and were based on indications at the time the list was 
published. As a result, discussions and conclusions could change significantly from 
issue to issue until final closeout. The changes reflected the progress in the analysis 
and resolution. Because of the frequency of the Apollo missions and the need for cor- 
rective actions before the next flight, this frequently updated format was a definite 
advantage. 

The Problem and Discrepancy List included problems from the problem -tracking 
list; problems identified during crew debriefings, particularly those pertaining to crew- 
operated equipment; and problems identified during the analysis of the in-flight photog- 
raphy and processed data. There was an average of 72 Problem and Discrepancy List 
items for each lunar -landing mission. 
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A report was published by MSC 30 days after each flight. In the 30 -day report, 
the anomalies were identified, the analyses discussed, and the corrective actions 
explained when determined. 

The mission report, published by MSC approximately 90 days after each mission, 
included sections in which each flight anomaly was discussed. Extracts from three 
mission reports are included in the appendix. An average of 35 anomalies were dis- 
cussed in each lunar -landing -mission report. 

Updating and publishing of the Problem and Discrepancy List was continued after 
publication of the mission report until all problems could be considered closed. Cor- 
rective action was always taken for critical failure modes. When the cause of a critical 
failure could be isolated to no less than two or more possibilities, corrective action 
was taken for each possible cause. A closed problem was usually one for which the 
cause had been determined and the corrective action was being implemented. A sepa- 
rate anomaly report was published for each anomaly that was not resolved and closed 
before publication of the mission report. Examples of anomaly reports are included in 
the appendix. 


Continuing Documentation 

Scientific data -gathering equipment and related communications and power equip- 
ment were deployed on the lunar surface by the crewmen on each of the six Apollo 
lunar-landing missions from July 20, 1969 (Apollo 11), to December 12, 1972 
(Apollo 17). The deployed equipment was designed to continue to provide data after 
return of the crewmen to Earth; therefore, from the standpoint of the experiments 
hardware specialist, the lifetime of each of these missions extends considerably 
beyond splashdown. 

Transmission of data from the Apollo 11 station ended on December 14, 1969. 
However, the other five stations were designed for considerably longer lifetimes, and 
communications with all five stations are continuing. Almost all the experiments that 
were deployed and activated continue collecting data, and analysis of the scientific data 
continues. 

The spacecraft -analysis/mission-evaluation action request form is being used 
to provide the documentation of communications between the flight controllers and the 
hardware engineers and scientists in much the same way as during the flight. Problem 
identifications, investigative actions and results, resolutions, and corrective actions 
are documented in this system. The Apollo Program Lunar Surface Equipment Status 
Report has replaced the Problem and Discrepancy List. This report is issued at approxi- 
mately 3 -month intervals; interim reports are issued if an earlier update is required 
for a specific purpose. 

Anomaly resolution documentation was handled in the same manner for the two 
particles and fields subsatellites that were launched into lunar orbit from the Apollo 15 
and 16 service modules. The Apollo 16 subsatellite mission ended May 29, 1972. The 
Apollo 15 subsatellite mission ended August 23, 1973. 
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CONCLUDING REMARKS 


The flight anomaly resolution system, developed and implemented during the 
Apollo Program, was used successfully for identifying and resolving problems and for 
determining the proper course of action within the relatively short time frame allowed 
by the Apollo Program schedule of missions. The success of this operation contributed 
to the overall success of the Apollo Program. The primary reasons for the degree of 
success achieved in flight anomaly resolution are as follows. 

1. The application of the concept of a mission evaluation team that was knowl- 
edgeable in all hardware systems serving as an integral part of the mission team 
while retaining the freedom to pursue problem solution to the extent required for 
mission and program success 

2. The selection of the proper personnel for this team, including the specialists, 
the team leaders, and the team manager 

3. The recognition that this team assignment was the prime activity for each 
member during the mission anomaly resolution period 

4. The establishment of an effective interface level for the evaluation team with 
other segments of the overall mission team 

5. The provision to the evaluation team of displays, data, and communications 
necessary for maintaining constant awareness of total system performance as the 
mission progressed 

6. The establishment of a definite point on which to focus direction and continuity 
of effort through a timely documentation system 

7. The implementation of a flexible updating system for problem resolution 
documentation to ensure a common understanding of problem status by members of the 
overall mission team, including project management 

8. The consistent application of the premise that, before any anomaly was con- 
sidered resolved, it was necessary 

a. That the details of how each affected system, component, or circuit func- 
tioned normally, and its purpose, be understood 

b. That the details of the anomalous condition be understood 

c. That the relationship of the affected function with other system functions 
and with the total mission be understood 
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d. That the corrective action be proved by test and/or analysis 

e. That the responses and documentation be reduced to the most practical 
and simple terms to ensure clarity and common understanding 


Lyndon B. Johnson Space Center 

National Aeronautics and Space Administration 
Houston, Texas, September 17, 1974 
914-89-00-00-72 
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APPENDIX 

EXAMPLES OF PROGRESSIVE DOCUMENTATION OF ANOMALY RESOLUTION 


Extracts from the anomaly resolution documentation system are included because 
the changes in the documentation from issue to issue reflect the progress made in the 
anomaly resolution process. The series of reports also illustrates some of the many 
conditions that influence the resolution process. The reports included are as follows. 

1. Extracts from Problem and Discrepancy List reports 

a. Apollo 12, December 5, 1969, GFE-1 

b. Apollo 12, December 16, 1969, GFE-1 

c. Apollo 14, April 12, 1971, ALSEP-18 

d. Apollo 14, May 14, 1971, ALSEP-18 

e. Apollo 14, July 15, 1971, ALSEP-18 

2. Extracts from mission reports 

a. Apollo 12 Mission Report, March 1970, Section 14. 3. 1 

b. Apollo 14 Mission Report, April 1971, Section 14.4.7 

c. Apollo 7 Mission Report, December 1968, Section 11. 21 

3. Complete Reports 

a. Apollo 14 Mission Anomaly Report No. 6, December 1972 

b. Apollo 7 Mission Anomaly Report No. 3, February 1969 

c. Apollo 10 and 11 Missions Anomaly Report No. 1, April 1970 

The four problems selected were among the less complex examples available. 
Resolution of the Apollo 12 lunar surface television camera problem involved examina- 
tion of the flight hardware, which was returned as a result of the problem. It was an 
obvious component failure and was resolved quickly. The problem hastened completion 
of the development of an improved image sensor. 

Resolution of the Apollo 14 active seismic experiment geophone problem required 
testing by telemetry uplink commands to the experiment and breadboard bench tests on 
the ground because the flight hardware had been deployed on the lunar surface. The 
problem occurred after completion of the flight and was not resolved before publication 
of the mission report. The progress made in the resolution process, including changes 
in the conclusions from issue to issue, is apparent in the extracts from the reports. 
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Resolution of the Apollo 7 command module floodlight problem involved examina- 
tion of the flight hardware as well as testing to determine expected lamp life and to 
develop proper preflight procedures. The flight hardware was available since it was 
part of the returned command module. 

As described in the anomaly report, the fuel cell condenser exit-temperature 
oscillations encountered on the Apollo 11 mission had been observed on several 
missions. Analysis in greater depth was required to understand the condition. The 
resolution included a detailed analysis of the flight data. 


APOLLO 12 LUNAR SURFACE COLOR TELEVISION CAMERA FAILURE 


The loss of the Apollo 12 color television picture occurred when the camera 
was inadvertently pointed toward the Sun while being transferred to the lunar 
surface from the modular equipment stowage assembly of the lunar module. The 
permanent loss resulted from both a crew operational procedure problem and a 
design weakness because the image sensor was susceptible to almost immediate 
damage if exposed to high -intensity light. Ground tests performed on similar 
equipment and on the returned hardware revealed the following. 

1. That light overstress from direct sunlight would immediately damage the 
image sensor 

2. That high- intensity light from highly specular reflections, as well as direct 
sunlight, would damage the image sensor 

3. That an image sensor more resistant to damage by high -intensity light was 
necessary 

4. That improved pointing procedures and controls were needed 

The cause of the problem was closed out in the documents that follow. The corrective 
action, which continued after the publication of these documents, produced a satis- 
factory image sensor and a remote-control cam era -pointing system for subsequent 
missions. 
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EXTRACT 


TIME: 

DESCRIPTION: 

DISCUSSION: 

IN PROGRESS: 


COMPLETION : 


Apollo 12 Problem and Discrepancy List 
December 5, 1969 


GEE-1 


During first extravehicular activity at 116 hours 

Lunar surface color television camera did not operate 
after deployment from lunar module. 

The camera provided satisfactory television coverage for 
approximately UO minutes. When moved to a different loca- 
tion, the picture failed when the tube was apparently ex- 
posed to direct sunlight or direct reflection from highly 
specular lunar module surfaces. 

Camera is now in Lunar Receiving Laboratory and will be 
released on December 6, 1969* After release, the following 
testing will be performed at MSC: 

1. Determine whether the color wheel is rotating; if pos- 
sible, without removing the camera case. 

2. Disable the automatic light control circuit and evaluate 
the total video signal to determine the extent of vidicon 
damage . 

3. Perform a mechanical/visual inspection of the equipment 
to determine if any other damage was incurred and if 
extreme heating had been experienced by the camera. 

This task will be completed within two weeks after receipt 
of camera. 

January 21, 1970 


ASSIGNED TO : Kingsley 



EXTRACT 


TIME: 

DESCRIPTION: 
DISCUSSION : 


Apollo 12 Problem and Discrepancy List 
December 16, 1969 


GFE-1 

During first extravehicular activity at 116 hours 

Lunar surface color television camera did not operate 
after deployment from lunar module. 

The camera provided satis factor:; television coverage for 
approximately Uo minutes. When moved to a different loca- 
tion, the picture failed when the tube was apparently ex- 
posed to direct sunlight or direct reflection from highly 
specular lunar module surfaces. 

After decontamination and cleaning, the camera was inspected 
and then operated. The picture that resulted was the same 
as that received at the end of the Apollo 12 transmission. 

The color wheel could be heard turning within the camera, 
and when the lens was removed, the turning was viewed 
directly. The power to the camera was turned on and off 
several times to see the wheel start and stop. By cutting 
one wire, the Automatic Light Level control circuit was 
disabled, and the camera turned back on. A well defined 
image was visible in the picture in the area which was 
formerly black. This black represented the undamaged area 
of the image sensor target. This test proved conclusively 
that the Apollo 12 failure was due to a damaged image sen- 
sor. 

In addition, the camera contractor deliberately damaged a 
similar image sensor by applying bright light levels. The 
resulting picture was similar to that of the Apollo 12 
camera. 

Subject to Apollo CCB approval on December 12, 1969, a lens 
cover will be provided for the Apollo 13 camera. Operational 
procedures and training will also be improved to reduce prob- 
ability of exposure. Automatic protective design approaches 
are also being evaluated. 
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EXTRACT 


Apollo 12 Mission Report 
March, 1970 

14.3 GOVERNMENT FURNISHED EQUIPMENT 


14.3.1 Color Television Failure 

The color television camera provided satisfactory television cover- 
age for approximately 40 minutes at the beginning of the first extrave- 
hicular activity. Thereafter, the video display showed only white in an 
irregular pattern in the upper part of the picture and black in the re- 
mainder. The camera was turned off after repeated attempts by the crew 
to restore a satisfactory picture. 

Ground tests using an Apollo-type image sensor (secondary electron 
conducting vidicon tube) exposed the camera system to extreme light 
levels. The resulting image on a monitor was very similar to that seen 
after the flight, camera failure. 

After decontamination and cleaning, the flight camera was inspected 
and power was applied. The image, as viewed on a monitor, was the same as 
that last seen from the lunar surface. The automatic light-level control 
circuit was disabled by cutting one wire. The camera then reproduced good 
scene detail in that area of the picture which had previously been black, 
verifying that the black area of the target was undamaged, as shown in 
figure 14-39. This finding also proved that the combination of normal 
automatic light control action and a damaged image-tube target caused the 
loss of picture. In the process of moving the camera on the lunar sur- 
face, a portion of the target in the secondary-electron conductivity 
vidiccn must have received a high solar input, either directly from the 
sun or from some highly reflective surface. That portion of the target 
was destroyed, as was evidenced by the white appearance of the upper 
part of the picture. 

Training and operational procedures, including the use of a lens cap, 
are being changed to reduce the possibility of exposing the image sensor, 
to extreme light levels. In addition, design changes are being considered 
to include automatic protection, such as the use of an image sensor which 
is less susceptible to damage from intense light levels. 

This anomaly is closed. 
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EXTRACT 




APOLLO 14 ACTIVE SEISMIC EXPERIMENT - ERRATIC DATA 
FROM NUMBER 3 GEOPHONE 


The number 3 geophone data from the Apollo 14 active seismic experiment became 
erratic. These data had been satisfactory during crew thumper operation on the lunar 
surface and also during high -bit-rate listening periods during the first 49 days after 
deployment at the lunar surface site. Tests of the emplaced flight hardware were per- 
formed by transmitting selected command sequences to the experiment, and the telem- 
etry data and experiment circuits were analyzed. The problem was duplicated by 
breadboard bench tests. Components taken from the lot that included the malfunctioning 
component were tested for a possible generic problem that would require correction 
before the next mission. 

The progressive steps in the analyses are apparent in the documents that follow. 
They reflect a better understanding of the cause as more results of tests and analyses 
became available. As indicated in the final anomaly report, the resolution also included 
an analysis of how the data from the number 3 sensor could be used if required for the 
analysis of a specific event. 
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EXTRACT 


Apollo 14 Problem and Discrepancy List 
April 12, 1971 

Italics indicate change from previous issue . 

PROBLEM AND DISCREPANCY LIST No . A&SEP-19 


Statement of Problem: 

Active seismic experiment no. 3 geophone data became erratic. 

Discussion’: 

On March 26 , the experiment was turned on in the listening mode (high bit 
rate) and Geophone #3 was spiking to off-scale high. The calibration command 
was transmitted to all three geophone channels . Geophone §3 channel went off- 
scale high simultaneous with the application of the calibration pulse and 
stayed off-scale high for the remainder of the listening period. During the 
one second period when the calibration pulse was present. Geophone #3 data 
showed four negative going pulses from off-scale high. Pulse widths were 
shorter than corresponding oscillations on Geophones 1 and 2 and decreased 
from the first to the fourth. The characteristics of these pulses indicated 
that Geophone #3 channel was operating at a higher gain than Geophones 1 and 
2 during the calibration period. Circuit analysis indicated that the problem 
had to be in the logarithmic compressor in the Geophone #3 amplifier chain. 

Signals are coupled into the logarithmic compressor through an input coupling 
capacitor which would block any dc voltage which would have to be present to 
drive the output voltage off-scale high for long periods (more than 1 or 2 
minutes) . 

The logarithmic compressor is basically an inverting amplifier with exponen- 
tial negative feedback . Two diode connected transistors between the output 

(see continuation sheet) 


Schedule: 


April 


15 


22 


29 


May 


Data Review 


Analysis 


Tests (Identify in Note} 


Changes (Identify in Note) 


Notes: 


1. Third lunar day data. 5 to 16 minutes of high bit rate data daily , 
April 4-12 

2. Preliminary analysis April 1 - 8 - Complete 

3. Bench checks and analysis. 


Personnel Assigned: 

J. Harris , Bendix, Geotech, Gadbois 


Conclusions: 









EXTRACT 


Apollo 14 Problem and Discrepancy List 
April 12, 1971 


Italics indicate change from previous issue . 


PROBLEM AND DISCREPANCY LIST 
CONTINUATION SHEET 


ALSEP-18 


Discussion: 


and input of the amplifier supply the feedback. The first diode is used 
for positive going and the second for negative going input signals. The 
diodes for all three geophone channels (2 per channel ) are physically lo- 
cated in an oven which controls their temperature at 105°C. 

Analysis indicates that the most probable cause of the problem is . an in- 
termittent open circuit in the diode feedback path. This would allow the 
amplifier input transistor to saturate, driving the output off-scale high. 
When signals large enough to drive the input stage out of saturation were 
present, the output would then respond and amplifier gain would not be 
compressed. 

Two copper paths conduct the feedback diodes to the logarithmic compressor 
amplifier. A solder crack in either path would then result in the data 
characteristics . Since the calibration voltage is applied to the 3 geo- 
phones by energizing 3 relays, the relay contact transfer shock could then 
disturb the solder crack and either open or close the feedback path. 

The problem was detected during the lunar night at the experiment location. 
Under lunar night conditions the thermal gradient between the oven (105°C) 
and the rest of the electronics (4.4°C) was maximum. Since thermal gra- 
dients cause mechanical stresses by differential thermal expansion, the 
large temperature gradient between the oven and the rest of the experiment 
electronics probably caused the crack. 

A breadboard of the logarithmic compressor will be constructed and the 
diode feedback loop will be opened to duplicate the experiment data. 
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EXTRACT 


Apollo 14 Mission Report 
April 1971 


lU.U.T Active Seismic Geophone 3 Electronic Circuit Erratic 

The experiment was turned on in the listening mode on March 26, 
1971,. and geophone 3 data were spiking off-scale high (fig. 14-34). 

When the geophone channels were calibrated, the geophone 3 channel went 
off-scale high simultaneously with the start of the calibration pulse 
and stayed off-scale high for the remainder of the listening period. 
During the 1-second period when the calibration pulse was present, data 
from geophones 1 and 2 showed the normal 7-hertz ringing caused by the 
calibration pulse. However, geophone 3 data showed four negative-going 
spikes coincident with the first four negative half cycles of the ring- 
ing on the other two channels. The spikes decreased in duration from 
the first to the last, the last having an amplitude of 90 percent of 
full scale (plus 2.5 volts to minus 2.0 volts). During the time that 
this pulse was present, the signal on channel 2 changed from minus 2.2 
volts to minus 2.35 volts, indicating that channel 3 was operating at 
an apparent gain of 30 times the channel 2 gain. 

As shown in figure 14-35, each geophone channel consists of a geo- 
phone, an input preamplifier, a low-pass filter, and a logarithmic com- 
pressor amplifier. The output of the logarithmic compressor feeds the 
instrumentation system. The logarithmic compressor is basically an in- 
verting amplifier with exponential negative feedback. Two diode-con- 
nected transistors between the output and input of the amplifier supply 
the feedback. The first diode is used for positive-going and the second 
for negative-going input signals. The diodes for all three geophone 
channels (two per channel) are physically located in an oven which con- 
trols their temperature at 105° C. 
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Figure 14-35*- Typical geophone ‘channel. 






EXTRACT 


Apollo 14 Mission Report 
April 1971 


It is believed that the failure is in the logarithmic compression 
amplifier because signals are coupled into it through an input coupling 
capacitor. This capacitor would block any offset voltages from the pre- 
ceding stages which would be required to drive the output off-scale high. 
Analysis indicates that the most probable cause of the problem is an 
intermittent open circuit in the diode feedback path. This would allow 
the amplifier input transistor to saturate, driving the output off-scale 
high. When signals large enough to drive the input stage out of satura- 
tion were present, the output would then respond and the output signal 
would not be compressed. 

The experiment electronics uses "cordwood" construction of the type 
which has caused solder cracks in other equipment. Two copper paths con- 
duct the feedback diodes to the logarithmic compressor amplifier, A 
solder crack in either path would then result in the data characteristics. 

There are 10 such solder joints for each geophone (fig. lU-36): 
four on the oven terminal board, four on the mother board, one on the 
top board of the log compressor module, and one on the bottom board of 
the log compressor module. The one most likely to fail first is on the 
top board of the log compressor module. Continuity at the joint re- 
covers as long as the crack closes during the lunar day. 


NASA-S-71-1704 



Figure lU- 36 .- Suspected cracked solder joints in amplifier. 
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EXTRACT 


Apollo 14 Mission Report 
April 1971 
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Figure Geophone calibration data. 


EXTRACT 


Apollo 14 Mission Report 
April 1971 


The log compressor modules for geophones 1 and 2 are of the same 
type construction. Since these are located slightly further from the 
oven than the one for geo.phone 3, the maximum temperature may not be 
quite as high. As a result, it may take longer for them to crack, if 
at all. 

Systems testing included operational thermal cycling tests over the 
temperature range for lunar day and night. However, cracked solder joints 
are a function of time as well as temperature, and apparently the ground 
test cycle did not allow enough time for a creep failure. This equipment 
was designed and built prior to the time when it was found that cordwood 
construction with soldered joints was unsatisfactory. 

A breadboard of the logarithmic compressor has been constructed, 
and the diode feedback loop will be opened to duplicate the experiment 
data. The mechanical design of the logarithmic compressor will be re- 
viewed to determine the 'changes that must be made to prevent solder 
cracks on Apollo l6. The active seismic experiment is not carried on 
Apollo 15* 

Procedural changes under consideration include operation of the 
oven to maintain compressor module temperature because the solder joint 
which is most likely cracked is in compression (stronger) at the higher 
temperature. 

This anomaly is open. 
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EXTRACT 


Apollo 14 Problem and Discrepancy List 
May 14, 1971 

Italics indicate change from previous issue . 

PROBLEM AND DISCREPANCY LIST 


Statement of Problem: 

Active seismic experiment no. 3 geophone data became erratic. 


No. ALSEP -18 


Discussion: 


On March 26, the experiment was turned on in the listening mode (high bit 
rate) and geophone 03 was spiking to off-scale high. The calibration command 
was transmitted to all three geophone channels. Geophone 03 channel went off- 
scale high simultaneous with the application of the calibration pulse and 
stayed off-scale high for the remainder of the listening period. During the 
one second period when the calibration pulse was present , geophone 03 data 
showed four negative going pulses from off-scale high. Pulse widths were 
shorter than corresponding oscillations on geophones 1 and 2 and decreased 
from the first to the fourth. The characteristics of these pulses indicated 
that geophone 03 channel was operating at a higher gain than geophones 1 and 
2 during the calibration period. Circuit analysis indicated that the problem 
had to be in the logarithmic compressor in the geophone 03 amplifier chain. 

Signals are coupled into the logarithmic compressor through an input coupling 
capacitor which would block any dc voltage which would have to be present to 
drive the output voltage off-scale high for long periods (more than 1 or 2 
minutes ) . 

The logarithmic compressor is basically an inverting amplifier with exponen- 
tial negative feedback. Two diode connected transistors between the output 

(see continuation sheet) 


Schedule: 


Data Review 


May 

1 8 


Complete 


Analysis 


Tests (Identify in Note) 


Changes (Identify in Note) 


IH 

IH 


ilil 

!§§ll 


Notes: i. 

Third lunar day data. 5 to 15 minutes of high bit 
April 1* - 12 

rate data daily. 


2 . 

Preliminary analysis April 1 - 8 - complete 



3. 

Bench checks and analysis 



Personnel Assigned: 



J. 

Harris, Bendix, Geotech, Gadbois 



Conclusions: 
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EXTRACT 

Apollo 14 Problem and Discrepancy List 
May 14, 1971 


Italics indicate change from previous issue . 

PROBLEM AND DISCREPANCY LIST alsep~i8 

CONTINUATION SHEET N °‘ 

Discussion: 


and input of the amplifier supply the feedback. The first diode is used 
for positive going and the second for negative going input signals. The 
diodes for all three geophone channels (2 per channel) are physically lo- 
cated in an oven which controls their temperature at 105°C. 

Analysis indicates that the most probable cause of the problem is an in- 
termittent open circuit in the diode feedback path. This would allow the 
amplifier input transistor to saturate, driving the output off-scale high. 
When signals large enough to drive the input stage out of saturation were 
present, the output would then respond and amplifier gain would not be com- 
pressed. 

The experiment electronics used ,f cordwood" construction of the type which 
has caused solder cracks in other equipment . Two copper paths conduct the 
feedback diodes to the logarithmic compressor amplifier. A solder crack in 
either path would then result in the data characteristics . 

There are ten such solder joints for each geophone: four on the oven ter- 

minal board 3 four on the mother board , one on the top board of the log com- 
pressor module and one on the bottom board of the log compressor module. 

The one most likely to fail first is on the top board of the log compressor 
module. Continuity at the joint recovers as long as the crack closes dur- 
ing the lunar day. 

The log compressor modules for geophones 1 and 2 are of the same type con- 
struction. Since these are located slightly further from the oven than the 
one for geophone 3 3 the maximum temperature may riot be quite as high. As 
a result 3 it may take longer for them to cracky if at all. 

A breadboard of the logarithmic compressor has been constructed and the diode 
feedback loop will be opened to duplicate the experiment data. 

The active seismic experiment is not on Apollo 15. 





EXTRACT 


Apollo l4 Problem and Discrepancy List 
July 15, 1971 


Italics indicate change from previous issue. 

PROBLEM AND DISCREPANCY LIST 


Statement of Problem: 

Active seismic experiment no. 3 geophone data became erratic. 


= Closure date. 


Mn. ALSEP-18 


CLOSED 


Discussion: 


On March 26, the experiment was turned on in the listening mode (high bit 
rate) and geophone no. 3 was spiking to off-scale high. The calibration 
command was transmitted to all three geophone channels. Geophone no. 3 chan- 
nel went off-scale high simultaneous with the application of the calibration 
pulse and stayed off-scale high for the remainder of the listening period. 
During the one second period when the calibration pulse was present, geophone 
no. 3 data showed four negative going pulses 'from off-scale high. Pulse 
widths were shorter than corresponding oscillations on geophones 1 and 2 and 
decreased from the first to the fourth. The characteristics of these pulses 
indicated that geophone no. 3 channel was operating at a higher gain than geo- 
phones 1 and 2 during the calibration period. Circuit analysis indicated that 
the problem had to be in the logarithmic compressor in the geophone no. 3 
amplifier chain. 

Signals are coupled into the logarithmic compressor through an input coupling 
capacitor which would block any dc voltage which would have to be present to 
drive the output voltage off-scale high for long periods (more than 1 or 2 
minutes ) . 

(see continuation sheet) 


Schedule: 

Data Review 

Complete 

Analysis 

Complete 

Tests (Identify in Note) 

Complete 

Changes (Identify in Note) 

None 



Notes: 1* Third lunar day data. 5 to 15 minutes of high bit rate data daily, 

April L - 12 

2. Mathematical analysis , May 11 

3. Bench checks and analysis . Breadboard tests , May 31 

4. Review > June 3 


Personnel Assigned: 

J. Harris, Bendix, Geotech, Gadbois 


Conclusions: Problem was caused by an intermittent failure (open) in the positive going 

diode wired transistor in the log compressor for the no. 3 geophone. Trans- 
istor failure appears to be an isolated case rather than a generic condition. 








EXTRACT 


Apollo 14 Problem and Discrepancy List 
July 15, 1971 


Italics indicate change from previous issue. 


PROBLEM AND DISCREPANCY LIST 
CONTINUATION SHEET 


No. alsep -18 


Discussion: 


The logarithmic compressor is "basically an inverting amplifier with exponen- 
tial negative feedback. Two diode connected transistors between the output 
and input of the amplifier supply the feedback. The first diode is used 
for positive going and the second for negative going input signals. The 
diodes for all three geophone channels (2 per channel) are physically lo- 
cated in an oven which controls their temperature at 105°C. 

Mathematical analysis and breadboard tests show that the cause of the prob- 
lem is an intermittent failure (open) in the positive going diode wired 
transistor . This allows the amplifier input transistor to saturate , driv- 
ing the output off-scale high. When signals large enough to drive the in- 
put stage out of saturation are present , the output responds , and amplifier 
gain will not be compressed. 

No previous failures of this type have been experienced during the develop- 
ment and test program. The collector of the intermittent transistor 
(2M2484-2B) is connected directly to the case without a lead. The emitter 
and base are connected from pin to chip by small aluminum leads which could 
possibly break and again make contact with temperature changes or by mechan- 
ical shock from nearby relay actuation. 

The experiment electronics uses ri cordwood n construction of the type that has 
caused solder cracks in other equipment however , a cracked solder joint in 
the cordwood construction , in this case, will not produce the flight re- 
sults. Apparently some inadvertent strain relief configuration in the cord- 
wood construction has averted a cracked solder joint, - at least, for the 
time being. 

The next mission scheduled for the active seismic experiment is Apollo 16. 
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ACTIVE SEISMIC EXPERIMENT GEOPHONE 3 DATA ERRATIC 


STATEMENT OF ANOMALY 


During the high-bit-rate listening mode period on March 26, 1971, the 
geophone 3 data showed intermittent spikes going off-scale high. When the 
calibration signal was applied to all three geophones, the geophone 3 data 
went off-scale high for the remainder of the listening period. A similar 
response was observed during the succeeding weekly listening mode periods. 


SYSTEM DESCRIPTION 


The active seismic experiment consists of an array of three geo- 
phones, a thumper with 21 initiators (detonators), a mortar package with 
four rocket-launched grenades, a central electronics assembly, and con- 
necting cabling. The three geophones are deployed on the lunar surface 
in a straight line at 150-foot intervals and are used to translate physi- 
cal surface movement into electrical signals. The thumper initiators, 
fired at intervals along a path parallel to the geophone line, and the 
mortar grenades, propelled to distances of 500 to 5000 feet away from the 
geophone line, artificially produce seismic waves which are detected by 
the geophones. In addition, the geophones monitor naturally produced lu- 
nar seismic activity in the 3 to 250 Hz range. 

The construction of a geophone is shown in figure 1. A coil of cop- 
per wire (seismic mass) is suspended in a magnetic field by flat canti- 
lever springs which also guide the coil vertically. The magnetic field 
is produced by a permanent magnet frame. Relative motion between the 
coil and the frame generates an output voltage across the coil, and this 
voltage is supplied through contacts of the calibrate relay (fig. 2) to 
the input of a preamplifier. The preamplifier output is then fed through 
a low-pass filter to a logarithmic compression circuit. Signals are cou- 
pled into this circuit through an input coupling capacitor which would 
normally block any d-c voltage that must be present to drive the output 
voltage off-scale high for long periods (more than 1 or 2 minutes). The 
output signal from the logarithmic compression circuit (the logarithm of 
the input signal) is supplied to the telemetry system. 

The logarithmic compression circuit provides adequate resolution when 
small signals are present, but also has the capability to respond to the 
expected 60- to 80-dB signal range. The circuit is basically an inverting 
amplifier with exponential negative feedback. Two diode-connected tran- 
sistors, in parallel, between the output and input of the amplifier supply 





the feedback signal (fig. 2). One of the transistors is used for the pos- 
itive-going input signals, and the other is used for the negativ.e-going 
input signals. The feedback transistors for all three geophone channels 
(two per channel) are physically located in an oven that maintains their 
temperature at 105° C. 

The geophone is calibrated by applying a fixed-amplitude, 1-second 
current pulse to the geophone coil. The current forces the coil to move 
to a new balance position. The coil will oscillate about the new posi- 
tion until the oscillations damp out. The geophone resonant frequency, 
sensitivity, and damping factor can then be determined from the damped 
oscillation waveform generated within the coil. 


DISCUSSION 


The active seismic experiment was initially turned on in the thumper 
mode with the central station in the high-bit-rate mode during lunar sur- 
face extravehicular activities on February 5, 1971. Vibrations from each 
of 13 thumper firings were received by each of the three geophones and 
the data were transmitted to earth. The experiment was then commanded 
to the standby mode, and was to remain in that mode except for 30 minutes 
of operation each week in the listening mode and the high-bit-rate mode. 
System operation was normal during the first six weeks; however, during 
the seventh-week monitoring cycle (March 26, 1971), at about lunar mid- 
night, the geophone 3 data were spiking off-scale high. 

All three geophone channels were calibrated, and the geophone 3 chan- 
nel went off-scale high simultaneously with the application of the cali- 
bration pulse (fig. 3). The channel stayed off-scale high for the remain- 
der of the listening period. During the 1-second period when the calibra- 
tion pulse was present, the geophone 3 data showed four negative-going 
pulses from off-scale high. Pulse widths were shorter than corresponding 
oscillations on geophones 1 and 2, and decreased in width from the first 
to the fourth pulse. The characteristics of these pulses indicate that 
the geophone 3 channel was operating at a higher gain than geophones 1 
and 2 during the calibration period, and that its output was biased off- 
scale high. 

A failure in the geophone, preamplifier, or filter could produce a 
bias offset; however, the compressor input coupling capacitor would block 
the bias offset and the output signal would not remain off-scale high. 

Had the input capacitor shorted, the output would be shifted, but not off- 
scale-high. The fault, therefore, must have been within the logarithmic 
compressor circuit (fig. 2). 
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Figure 3.- Geophone calibration data on March 26, 1971* 


The failure was duplicated by opening the positive-going diode-wired 
transistor. This saturated the amplifier input transistor, and drove the 
output off-scale high. When signals large enough to drive the input stage 
out of saturation were present, the output responded and the amplifier 
gain was not compressed until the output voltage was driven to zero. The 
part of the signal which went below zero voltage was then compressed by 
the negative-going feedback diode and the narrow pulses shown in figure 3 
were produced. 

The transistor emitter and base are connected from pin to chip by 
small aluminum wires (fig. 4.). A break in one of these wires, or a par- 
tial or complete bond failure between the wire and the chip would cause 
the open circuit. The elements could also make contact again to produce 
the intermittent condition when mechanically shocked by nearby relay 
actuation. 

No failures of this type occurred during the development and test 
program. To determine if a generic defect existed, a group of ten tran- 
sistors from the same lot were examined internally and subjected to ther- 
mal shock and bond pull tests as well as electrical tests. The transis- 
tors were tested within the manufacturer's specification limits and also 
beyond specification limits and were found to be satisfactory. 

With a significant event, such as firing the mortar, the negative 
side of geophone 3 is capable of distinguishing the arrival of the event 
signal. Calculations of new circuit constants for an open in the posi- 
tive-going transistor make the channel usable for interpretation of dis- 
cernable negative data. 

The experiment was flown on Apollo 16 without any changes to the ac- 
tive seismic detection system, and has performed without a recurrence of 
the Apollo 1^ problem. 


CONCLUSIONS 


An intermittent failure (open) most likely occurred in the positive- 
going diode-wired transistor in the logarithmic compressor for geophone 3, 
resulting in the intermittent loss of data from the geophone. 


CORRECTIVE ACTION 


No corrective action was taken since the history of the transistor 
indicated that a generic problem did not exist. 




APOLLO 7 FAILURE OF LOWER EQUIPMENT BAY FLOODLIGHTS 


The loss of the primary floodlights in the Apollo 7 command module lower equip- 
ment bay was an example of crew observation of an off-nominal condition as well as of 
a preflight procedure problem reflecting a lack of understanding of the life-limiting 
details of some components. The flight hardware was mounted in the returned command 
module and thus was available for examination and testing. The procedures employed 
in determining the cause and the necessary corrective action are apparent in the reports 
that follow. Extracts from Problem and Discrepancy List reports are not included 
because at the time of the Apollo 7 mission, this part of the reporting system had not 
yet been formalized and did not extend past the time of the mission report. 


40 



EXTRACT 


Apollo 7 Mission Report 
December, 1968 

11 . 21 FAILED FLOODLIGHTS 


Sometime during the mission, both of the primary lamps failed in 
the lower equipment bay floodlights. Postflight investigations revealed 
that the lamp filaments (cathodes) had completely vaporized, which caused 
a diode to short in each lamp driver. 

A new lamp has a start-up voltage of about 500 volts. As the lamp 
ages, the cathode deteriorates, thus increasing the start-up voltage, 
which can go as high as 1800 volts. The diode is rated at TOO volts; 
therefore, it would burn out. The rate of cathode deterioration is de- 
pendent on the operating voltage. Maximum deterioration rate occurs when 
the dimming rheostat is halfway between the full-dim and full-bright posi- 
tions. 

Tests are in progress to establish lamp life at the critical opera- 
ting voltage. Normally, these lamps should operate 2000 hours. 

Procedural changes are being made to use only the secondary lamp 
on full bright during ground tests, and consideration is being given to 
installing flight lamps just prior to the countdown demonstration test. 

No hardware changes are planned. This anomaly is still open. 
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FAILURE OF LOWER EQUIPMENT BAY FLOODLIGHTS 


STATEMENT 


Both of the primary lamps in the lower equipment bay floodlights 
failed during the mission. 


DISCUSSION 


Two floodlights, mounted on the X-X foot struts, provide illumina- 
tion of the work areas and panels in the lower equipment bay. Each 
floodlight fixture consists of a primary and secondary fluorescent lamp 
and a lamp driver circuit (fig. l). 

The primary lamps in both fixtures failed during the mission. Post- 
flight investigations revealed that the filament (cathode) in each lamp 
had vaporized, shorting a diode in the associated lamp driver circuit. 

A new lamp has a starting voltage of about 500 volts, but as the lamp 
is used, the cathode is expended. As the lamp approaches its life limit 
of 2000 hours, the starting voltage increases and may reach values as 
high as 1800 volts; therefore, the diode in the lamp driver circuit, 
rated at TOO volts, could short. 

The rate of cathode expenditure is related to the operating voltage, 
with the expenditure rate being maximum when the dimming rheostat is 
about halfway between the dim and full-bright position. The _ floodlights 
on the Apollo 7 spacecraft had a minimum of U00 hours of use prior to 
launch, and the operating intensity was not determined. The Apollo 8 
spacecraft had approximately 100 hours of use prior to launch, and no 
failures were experienced during that mission. 

Tests were performed with a new lamp operated at a critical voltage 
level for maximum cathode expenditure . Although the cathode heaters 
failed after 80 hours, the lamp could be turned off, restarted, and dim- 
med until the test was terminated after 6U8 hours. 


CONCLUSION 


The primary lamps probably failed because of excessive operation 
time in the dimmed position prior to flight. 












APOLLO 10 AND 11 FUEL CELL CONDENSER EXIT-TEMPERATURE OSCILLATIONS 


Fuel cell condenser exit-temperature oscillations had been observed on several 
missions. When they recurred during the Apollo 10 mission, the need for an in-depth 
investigation to better understand the mechanism and the significance of the oscillations 
was apparent. In this example, the flight hardware could not be returned for examina- 
tion because it was mounted in the service module. The service module separates from 
the command module at the time of entry and is not recoverable. 

The anomaly resolution analysis procedures and the results are presented in the 
report that follows. Extracts from Problem and Discrepancy Lists and mission reports 
are not included because the anomaly resolution applies to several missions and because 
the investigation was subsequent to individual mission-oriented activities. 
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FUEL CELL CONDENSER EXIT TEMPERATURE OSCILLATIONS 


STATEMENT OF ANOMALY 


During the Apollo 10 flight, the condenser exit temperature for 
fuel cell 2 experienced limit-cycle oscillations of from 15° to 20° F 
full amplitude while in lunar orbit. Review of the flight data revealed 
the presence of a periodic disturbance throughout the flight, and analy- 
sis showed this disturbance was the trigger for the oscillations. The 
disturbance was typified by a rapid 1.5° to 2° F drop in condenser exit 
temperature with a slow recovery to the initial value and occurred every 
3 to 8 minutes (fig. l). The Anomaly Summary Section of the Apollo 10 
Mission Report (MSC-00126) describes the disturbances and oscillations 
in detail. 

Fuel cell 2 condenser exit temperature was also periodically dis- 
turbed during the entire Apollo 11 flight (fig. 2) but did not experience 
the oscillations observed in Apollo 10. A further review of available 
data shows the presence of a similar disturbance on either fuel cell 2 
or 3 (one affected fuel cell per flight) during Apollo 7, 8, and 9* As 
a result, an in-depth investigation was conducted to identify the cause 
of the disturbance and its significance in terms of future inflight fuel 
cell performance. Three essentially independent investigations of these 
disturbances were performed by North American Rockwell, Pratt and Whitney, 
and the NASA Manned Spacecraft Center. The findings of these investigators 
were discussed in detail in meetings at the Manned Spacecraft Center on 
September 11 and 12, 1969, and the combined results of these analyses, 
are presented herein. 

DISCUSSION 


An intensive analysis of the available data, particularly from 
Apollo 10 and 11, resulted in the following observations as to the dis- 
turbance and its cause. 

a. Apollo 11 data indicate the secondary bypass valve (fig. 3) did 
not move as a result of the periodic disturbance; this observation is 
based on the slow recovery time (on the order of h minutes) for condenser 
exit temperature following each disturbance. If the valve had moved, full 
recovery should have taken place in a matter of seconds . The Block II 
retrofit secondary coolant bypass valve (Block I valve poppet) may have 
contributed to this behavior; however, the valve characteristics do not 
support this conclusion, since the response times of the two valves 
(Block II and Block II retrofit) are virtually the same for a condenser 
exit temperature disturbance of the type experienced in flight. 


b. Apollo 10 data indicate the secondary coolant bypass valve did 
respond to the disturbance, based on the rapid condenser exit temperature 
recovery rate after each disturbance, the slight condenser exit tempera- 
ture overshoot observed on most of the Apollo 10 disturbances and the fact 
that oscillations were observed. 

c. Apollo T, 8, and 9 data indicate the presence of a disturbance 
throughout each flight with period and amplitude similar to the disturb- 
ance observed on Apollo 10 and 11. 

d. Apollo 11 data indicate the condenser exit temperature disturb- 
ance characteristic was either altered or its presence was questionable 
during and immediately after major burns of significant duration, indi- 
cating a sensitivity to longitudinal acceleration. 

e. The condenser exit temperature disturbance has only been observed 
under zero-g flight conditions. 

f. Fuel cell performance was not affected by the condenser exit tem- 
perature disturbance nor, in the case of Apollo 10, by the resultant oscil- 
latory behavior. 

g. Only one fuel cell, either in position 2 or 3, has experienced 
the disturbance on each flight. 

h. Apollo 10 and 11 data show a definite relationship between the 
frequency of the disturbance and the electrical load on the affected fuel 
cell (fig. U). 

Each of the following items was investigated prior to a detailed 
analysis of the possible causes of the disturbance and each produced neg- 
ative findings. 

a. Variations or similarities of secondary loop (water/glycol) 
plumbing for all affected fuel cells , including line routings , loop 
volume, and component calibrations 

b. Variations or similarities of all component and powerplant 
acceptance and checkout test results of affected fuel cells , including 
line routings, loop volume, and component calibrations 

c. Variations or similarities of the physical properties (surface 
tension and viscosity) of Type II water/glycol solutions from different 
lots 


d. Variations or similarities of fuel cell and spacecraft inter- 
face hardware configuration regarding affected fuel cells . 
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Subsequent to Apollo 11, all possible causes of the disturbance were 
listed to serve as a basis of investigation for the three analytical teams 
The disturbance characteristics were also listed, and each possible cause 
was examined for conformance with these characteristics. Table I summar- 
izes the possible causes and characteristics. A discussion of each of 
these possible causes is presented in the following paragraphs. 


Possible Causes Eliminated 

The following possible causes of the temperature oscillations were 
determined to be invalid. 

Slug of cold water /glycol from the radiator .- A slug of cold water/ 
glycol from the radiator with a radiator outlet temperature drop rate 
greater than 50° F./minute could cause the condenser exit temperature dis- 
turbance. As the slug entered the condenser, a drop in the condenser 
exit temperature on the order of 2° F would result. However, there is 
no reason to expect a slug of cold water/glycol to occur periodically in 
the radiator and exhibit a period which varies with load as observed on 
Apollo 10, nor would this cold slug explain the slow condenser exit tem- 
perature recovery observed on Apollo 11. Since no indication of such a 
radiator response was observed on any of the affected fuel cell coolant 
loops , this possible cause was eliminated. 

Blockage of secondary regenerator cold side (hot side of secondary 
coolant bypass valve ).- Blockage of the cold side of the secondary re- 
generator, as might result from bubble or sludge entrapment, could pro- 
duce the disturbance in the condenser exit temperature. Such blockage 
would result in an incorrect hot /cold mixture ratio at the secondary by- 
pass valve, with the colder fluid from the- radiator reaching the conden- 
ser and causing a loss of coolant regeneration capability. This type 
of blockage is unlikely, since, to exhibit the proper rate of decrease 
in the condenser exit temperature, it must occur suddenly with flow 
blockage greater than 90 percent. In addition, the passages would have 
to be opened slowly to exhibit the recovery characteristic noted after 
the Apollo 11 disturbance. Bubble entrapment would be expected to build 
up slowly and exhibit a rapid recovery as the bubble passed through the 
regenerator. The periodic variation could possibly result from the sludge 
or bubble circulating around the loop. Only a minor change in the dis- 
turbance period, would be expected with changing load (full regenerator 
versus full bypass flow), on the order of 1 minute difference between the 
two flow modes. The period variation with load observed on Apollo 10 was 
approximately U to 8 minutes and cannot be explained by regenerator cold 
side blockage. 


Sticking secondary coolant bypass valve .- A secondary bypass valve 
which was restricted in movement could possibly cause a condenser exit 
temperature disturbance, since an incorrect mixture ratio could cause a 
cold slug of water/glycol to reach the condenser. There is, however, no 
apparent reason for the regularity unless sludge is circulating in the 
loop. A sticking valve would also require a corresponding change in 
electrical load or radiator outlet temperature in order for the bypass 
valve to be incorrectly positioned near the zero regeneration point. This 
possible cause requires that the valve be incorrectly positioned and re- 
stricted in travel at the time of each disturbance. Since there is no 
reason for the valve to ever reach the necessary position under the ob- 
served operating conditions, this possible cause was rejected. 

Temporary blockage of coolant (water/glycol) side of condenser . - 
Temporary condenser blockage (20 to 25 percent) on the coolant side, as 
with a bubble or sludge entrapment, could produce the disturbance. Again, 
the periodicity of the occurrence could be explained by the bubble or 
sludge circulating through the loop, if the apparent load dependency as 
noted on Apollo 10 and 11 is discounted. However, the condenser exit 
temperature should increase slightly as the effective condensing area is 
reduced and then drop as the core becomes unblocked when the contaminant 
moves through. An increase in condenser exit temperature prior to the 
disturbance has not been consistently observed, with only a few such in- 
stances on Apollo 10 which are considered to be within the noise level of 
the data. Additionally, test and performance experience has shown the 
secondary bypass valve to be sooner affected by contamination than other 
coolant loop components . 

Hydrogen pump shutdown and restart .- Intermittent stop-start oper- 
ation of the hydrogen pump can exhibit the proper temperature disturbance 
characteristic. Testing has demonstrated extremely rapid drops in con- 
denser exit temperature for both pump shutdown and startup. However, the 
regularity of the observed disturbance is inconsistent with a randomly 
intermittent hydrogen pump which most probably would fail completely early 
in the flight. Also, there is no reason why the pump would operate in 
this mode in zero-g operation only. 

Coolant flow variations .- A slow decrease in coolant flow followed 
by an abrupt increase, such as might result from filter plugging and un- 
plugging, could produce the characteristic disturbance. Periodicity 
might be explained by contaminant circulation. A coolant flow increase 
of 20 to 30 pounds per hour would cause a 2° to 3° F condenser exit tem- 
perature change. However, this change would be observable in radiator 
delta temperature (inlet/outlet), a condition which was not observed. 
Variations of the period with load are also not explained for this pos- 
sible cause. 



Water discharge blockage .- Loss of water discharge capability, as 
could be caused by high water back pressure or an intermittently stick- 
ing discharge valve, was considered as a possible cause of the disturb- 
ance. This cause, however, would not be expected to demonstrate the ob- 
served disturbance characteristics. The primary regenerator hot -side 
temperature and condenser inlet temperature would be reduced as the water 
flashed to steam in the regenerator. More significantly, rapid tempera- 
ture changes in the hydrogen loop would be damped before the effects 
could be observed in condenser exit temperature changes , largely because 
of the overwhelming thermal capacitance of the primary coolant fluid sys- 
tem. A high water back pressure would be manifested in all three fuel 
cells because of the common water discharge manifolding. Additionally, 
a "pH Hi" indication would be expected, as was observed during a thermal 
vacuum test when the water discharge line froze and blocked. No "pH Hi" 
indications were observed during any flight when the condenser exit tem- 
perature disturbance was present. 

Erratic primary bypass valve .- Erratic control by the primary bypass 
valve could affect condenser exit temperature, since a sudden opening 
would tend to increase condenser exit temperature and a sudden closure 
would lower the exit temperature. However, recent tests have shown that 
hydrogen loop temperature perturbations in the vicinity of the cell stack 
are well damped prior to reaching the condenser. Also, an erratic pri- 
mary valve would not be expected to produce a periodic disturbance, nor 
is there any reason why such behavior would not also be observed during 
ground operation. 

Restricted secondary coolant bypass valve flow .- Temporary blockage 
or restriction of flow through the secondary bypass valve followed by 
sudden release could produce the observed disturbance characteristic. 

Such blockage could be caused by circulating contamination, probably in 
a gelatinous form. Experience has shown that contamination in the coolant 
loop is more likely to collect in the vicinity of this valve (as during 
Apollo 7 and 9) than anywhere else in the loop. In theory, this mass of 
contamination would periodically accumulate in the vicinity of the valve, 
generally in the bypass flow passage of the valve and, when released, 
would cause a sharp decrease in condenser exit temperature because of the 
suddenly increased bypass fluid flow. 

This phenomenon is considered to be a rather remote possibility for 
two reasons: (a) if the bypass flow were slowly being restricted, as it 

would probably be for contamination in the loop, an increase in con- 
denser exit temperature above the normal operating value should be ob- 
served, but this was not the case unless it is assumed that the low con- 
denser exit temperature (at the lower end of the disturbance) was the 
steady operating value corresponding to the valve position, and (b ) the 
relationship of the disturbance period to load cannot be explained. Also, 


the similarity of the disturbance among the five affected fuel cells tends 
to negate the contamination theory, since circulating contamination would 
not be expected to produce nearly identical results on each affected 
powerplant. On Apollo 7 and 9> slow condenser exit temperature excursions 
to abnormal values (not periodic disturbances) were attributed to re- 
stricted secondary coolant bypass valve travel resulting from loop con- 
tamination. However, none of the fuel cells with restricted bypass valves 
displayed any evidence of a periodically disturbed condenser exit tem- 
perature. Additionally, the Block II retrofit valve on Apollo 11 was 
known to be less sensitive to contamination than the standard Block II 
valves but was affected in the same manner as the Block II valves ; this 
fact implies that some phenomenon other than contamination is the cause 
of the disturbance. 


Most Probable Cause 

One possible cause remained that could not be eliminated from the 
list; that is, condenser water retention and periodic release (a water 
slug from the condenser). The water slug meets more of the criteria 
that describe the condenser exit temperature disturbance than does any 
other possible cause. 

Condenser water retention on the hydrogen side .- Blockage of the 
condenser core on the hydrogen side could produce the proper disturbance 
characteristics. Condensed water would have to collect and be contained 
either in or near the condenser exit by surface tension forces. This 
water would then be released periodically in fairly large quantities as 
the accumulated water became sufficient to overcome retaining forces. If 
the water were retained at the core, the temperatures would essentially 
equal the water/glycol inlet temperature. The subcooled water reaching 
the temperature probe would cause the temperature drop indicated In the 
data, and, if the water were retained on this probe because of surface 
tension until gradually blown off or evaporated, the slow recovery char- 
acteristic observed in flight could be explained. Since the condenser 
exit temperature sensor is downstream of the bypass valve Vernatherm 
sensor, the released slug of water might also impinge on the valve sensor 
and, if large enough and cold enough, would cause valve motion. Periodic 
recurrence would result from the accumulation of water in the condenser 
as a result of surface tension until some critical level was reached and 
the water was again released. Dependence of the disturbance period on 
electrical loads would result from water production increasing with in- 
creasing load. The rate of change of water condensation is known to bear 
a near-instantaneous response relationship with step load changes. 

Examination of sequence photography of the hydrogen exit plenum, with 
the condenser operating in a horizontal position during ground tests, has 


shown that surface tension plays a large part in removal of water conden- 
sate, even in one-g operation, and that the water does not leave the con- 
denser core in a uniform fashion. Normal one-g condenser operation is in 
a vertical position relative to the launch pad. In these films, relative- 
ly large quantities of water were retained in some of the hydrogen pas- 
sages and on the coolant passage walls at the end of the condenser core 
by surface tension and frequently joined together before being released 
as large globules into the exit plenum (fig. 5). A definite variation 
was observable in wetting characteristics across the end of the condenser. 
These results, along with the fact that a slug of water out the condenser 
can produce all of the characteristics of the disturbance in zero g, make 
this possible cause the most likely candidate of those considered. 

The question of why the disturbance has not been present on every 
fuel cell if a water slug is the cause remains to be answered. Analyses 
indicate that variations in condenser passageways (hydrogen side), surface 
wetting properties, and/or hydrogen loop flow rates between individual 
powerplants allow this phenomenon to take place only on those particular 
powerplants where combinations of these variables are exactly right. In 
•fact, figure k illustrates a difference in characteristics between two 
affected , powerplants . From the visual data available in the condenser 
test films, it is reasonable to assume that water does not depart the 
condenser in uniformly sized droplets on any powerplant but rather, in 
zero g, comes off the condenser in globules of varying sizes at intervals 
which depend on such factors as surface tension (wettability), loop flow 
rate, condenser passage area (cross-section), braze uniformity, etc. If 
it is further assumed that a critical combination of these conditions is 
required to produce a water slug of a magnitude sufficient to manifest 
an observable condenser 'exit temperature disturbance, then it is likely 
that only a certain number of fuel cells will exhibit the disturbance. 

Of the 15 Block II fuel cells flown to date, five have displayed the 
characteristics. If a water slug is the cause of the disturbance, there 
is no reason to expect a change in the ratio of affected fuel cells, 
since no process changes were made. 

The appendix to this document presents the results of an analysis 
of the probable condensed water behavior and its effect on condenser exit 
temperature . 


CONCLUSIONS 


From all indications, the Apollo 11 fuel cell 2 
valve (a Block II retrofit) did not move as a result 
condenser exit temperature disturbance. 


secondary bypass 
of the periodic 



If the cause of the disturbance is a water slug from the condenser, 
the incidence rate of the condenser exit temperature disturbance should 
not change significantly. 

Neither the temperature disturbance nor the temperature oscillations 
have shown any degradation in fuel cell performance. No hardware changes 
will be made; however, procedures have been developed to effectively damp 
these oscillations, should they occur. 


TABLE I.- POSSIBLE CAUSES AND CHARACTERISTICS OF DISTURBANCE 


Characteristics of disturbance 


1. Periodic 

2. Occurred only under zero-g conditions altered or eliminated by major 
propulsive maneuvers 

3. Condenser exit temperature typically dropped 1.5° to 3.0° F in 2 to 
10 seconds 

U. Period of disturbance related to fuel cell load 

5. Rapid condenser exit temperature recovery for Block II bypass valve 
(<20 seconds) 

6 Slow condenser exit temperature recovery for Block II retrofit by- 
pass valve (~U minutes) 


Possible causes 

1. Cold water/glycol slug from radiator 

2. Secondary regenerator cold side blockage 

3. Secondary coolant bypass valve motion restriction (sticking) 
h. Condenser coolant blockage 

5. Intermittent hydrogen pump operation 

6. Intermittent coolant pump operation 

7. Water discharge blockage 

8. Erratic primary bypass valve 

9. Slugging of condenser water 

10. Restricted secondary coolant bypass valve flow 







Fuel cell schematic 



Average current, A 


Figure U.- Load- period relationship 
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Figure 5.- Sequence photographs of hydrogen 
exit plenum. 
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(a) 



(e) 


Figure 5 • - Concluded. 
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APPENDIX 


Water Condensate Behavior Analysis 

Several theories have been investigated to explain the abnormal 
variation of primary side condenser exit temperature. The most promis- 
ing of these is a water buildup at the condenser with resultant slug 
flow of subcooled condensate downstream to the temperature sensor and 
the Vernatherm sensing element of the secondary bypass valve. 

By calculating an effective radius of the core passages in the 
.condenser and assuming perfect wetting, a balance of surface tension 
and pressure forces shows that a pressure differential of approximately 
0.02 psi (1.96 cm H 2 O) is required to clear one condenser core passage 
after it has been filled with water. A test performed to measure the 
gas pressures needed to blow the water out showed the average required 
pressure differential to be 1.65 cm H 2 O. This test was repeated on 
1+5 tubes, and the extremes ranged from 1.55 to 1.75 cm H 2 O. The fric- 
tional pressure drop, on the other hand, was calculated for normal con- 
ditions (clean passage) to be approximately 0.002 psi, or an order of 
magnitude less than the pressure drop required to overcome capillary 
forces. This would lead to the hypothesis that the small trapezoidal 
passages within the condenser core fill up randomly (due to the random 
nature of condensation sites) until sufficient pressure head is built 
up to overcome the capillary forces, at which time the slug is blown 
free. Sequence photography proves that this does indeed occur. 

A cross-section of condenser core geometry is shown in figure A-l. 

The coolant and condensate sections alternate, leaving a solid end-strip 
of metal between each condensate layer. In a one-g test, sequence photog- 
raphy shows that as the condensate accumulates at the tube ends , some 
tubes become totally filled with water. When enough tubes become filled, 
dynamic pressure forces push the expanding water globules out of the 
condenser and into the exit plenum area. Even after the surface tension 
forces within the tube are overcome, the additional capillary action of 
the end strips tends to accumulate condensate on these strips. This ac- 
cumulated condensate could not run off easily in the one-g test, due to 
the barrier created by brazing the end plates to the core. Hence, even 
in one g, all end strips can become completely covered with water during 
normal operation with the condenser in a horizontal position. In zero g, 
this effect would be even more pronounced, allowing much thicker layers 
of water to be built up on the end of the condenser. The thickness of 
these layers would build up until the combination of random pressure 
fluctuations in the gas stream and viscous shear stresses at the water- 
gas interface become sufficient to force globules of condensate from the 
condenser. A test showed that very large globules can accumulate, even 



in a one-g environment. An attempt was made to calculate the size of 
these globules and to correlate their transport frequency with fuel cell 
load. This attempt was futile because the pressure fluctuations are 
completely random, and the variations in core geometry due to brazing and 
fin shape distortion are impossible to calculate. 

Large condensate globules leaving the condenser collect on the solid 
end strips at the glycol inlet to the condenser and could be subcooled 
as much as 20° F below the normal l60° F saturation temperature if allowed 
to remain long enough. In zero g, when a subcooled globule leaves the 
condenser, most of it will be clinging to the walls of the transfer tube 
(upper view of fig. A-2 ) , since it will tend to collect in the exit mani- 
fold as shown in the lower view. It will then proceed downstream until 
it comes to the enlarged area for the Vernatherm, or valve sensing ele- 
ment (fig. A-3). If the condensate layer is sufficiently thick, contact 
will be made between the Vernatherm fins and the condensate, and the wet- 
ting properties of the fin-tube geometry will cause the condensate to 
cover, or attempt to cover, the entire exposed surface area of the Verna- 
therm element. If, on the other hand, the condensate layer is not suffi- 
ciently thick to cause adhesion to the Vernatherm element, it will be 
forced downstream by the viscous shearing stresses at the liquid-gas inter- 
face until it reaches the temperature sensor. Average clearance between 
the end of the temperature sensor probe and the tube wall is 0.19 (±0.02) 
inch. This clearance can vary considerably, however, with positioning of 
the probe angle at assembly. Vernatherm element-to-wall clearance is. 
typically 0.1 inch. As at the Vernatherm, if the condensate layer is suf- 
ficiently thin, it will not touch the sensor. If, however, the layer ex- 
ceeds a certain critical thickness, the capillary action of the condensate 
on the sensor will cause the water to almost immediately engulf the sensor 
probe (fig. A-3, lower view), which is very much smaller than the Verna- 
therm element. It should be emphasized that each fuel cell will have a 
different critical thickness between the tip of the sensor and the tube 
wall, depending on the tolerances at the mounting assembly of the sensor. 

The following is a proposed mechanism to explain the anomalous con- 
denser exit temperature behavior. In zero g, for a fuel cell with a very 
small critical thickness at the temperature-sensing element, subcooled 
condensate buildup occurs until a globule of water moves downstream of 
the condenser. Assume that the thickness of the condensate layer is not 
sufficient to cause adhesion at the Vernatherm element, due to the en- 
larged flow area for the Vernatherm, but is thick enough to engulf the 
temperature sensor. For the extreme case of 20° F subcooled condensate, 
the sensor temperature will immediately begin dropping at the rate of 
about 5° F/sec, which is greater than the rate seen in flight. Depending 
on the size of the liquid globule, the temperature will drop until the 
dynamic pressure forces of the gas stream blow it off the sensor, or 
until the heat transfer between the gas stream and the water-coated sen- 
sor allows the sensor probe to begin its recovery. After the condensate 


is blown off, the temperature recovery is much slower than the drop, due 
to the much smaller heat transfer coefficient between the gas and the 
sensor. The detected variation in condenser exit temperature will be 
similar to that shown in figure A-U (upper view). The temperature drop 
and subsequent recovery is on the order of seconds ; whereas the time 
between disturbances varies inversely with the current and is on the 
order of several minutes . 

For this case the temperature sensor will produce telemetry data 
showing temperature variations, but in reality the valve poppet posi- 
tion, and hence the bypass ratio, never changes. 

In the case of a service propulsion firing, a momentary gravity 
field is created in a direction coincident with the downward flow of 
condensate through the. condenser. This, in effect, cleans the tube 
walls by flushing the condenser entirely of condensate. The resulting 
very large subcooled condensate globule affects not only the temperature 
sensor but also the Vernatherm element and causes the valve to react ac- 
cordingly. The resulting variation in condenser exit temperature is 
shown in figure k-h (lower view). 
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Figure A-2.- Primary side condenser exit geometry. 








